An IEEE 1599-Based Interface for Score Analysis

This paper deals with a software application to visualize score analysis. Such a tool allows both the enjoyment of music in general, thanks to the synchronized execution of heterogeneous multimedia objects, and the highlighting of the results of score analyses. As a consequence, the user can investigate the compositional process and the relationships among the music objects of a given piece while listening to an audio track and viewing the corresponding score. This is made possible by a new XML-based standard for music representation – namely IEEE 1599 – which will be shortly described. 1 Score Analysis and Its Visual Representation Musical analysis embraces all those methods aiming at the comprehension of music processes and relationships within a composition. Of course, many different approaches have been employed and are in use to analyze a score, ranging from the traditional methods used in the past centuries to more recent techniques such as the functional analysis by Hans Keller and the Schenkerian analysis by Heinrich Schenker. A number of ongoing researches are aimed at the analysis of music information through algorithmic processes, in order to obtain automatic segmentation, recognition of recurrent patterns, statistical data, etc. However, the problem of how such results can be obtained, either in a hand-made or computer-based way, is not pertinent to this work. Rather, we concentrate on the computer-based techniques to produce and enjoy the visual representation of musical analysis. The recipients of this kind of applications include musicologists, music students and untrained people interested in understanding music mechanisms. Thanks to the application we will introduce in the next sections, we propose a new way to investigate music processes. The results coming from an analysis are presented to the user synchronized with a number of multimedia objects which describe the piece from other points of view. Thus, analysis is not a static description, but it is explained while the score is advancing and the corresponding audio track is playing. As a consequence, an overall comprehension of music can be provided, as regards its symbolic, structural, audio and graphical aspects. The problem we address has two faces: i) a phase of preparation which includes the encoding of the piece, the recognition of music structures to highlight, the assembly of An IEEE 1599-Based Interface for Score Analysis 273 multimedia objects referable to the given piece, and finally the choice of a format to describe all this information within a unique framework; ii) the design and implementation of interfaces to enjoy music according to the key concepts we have introduced. Even if the choice of a format to represent music and all the related materials is not the main subject of the present work, it should be evident that only a standard with some distinctive characteristics can provide an integrated representation of heterogeneous music contents. Consequently, such ad hoc standard is the base for an evolved interface to enjoy music and the corresponding structural information. Needless to say, popular standards for symbolic scores, graphics, audio, video already exist and are commonly used. Nevertheless, their use for a “comprehensive” description of music presents some disadvantages, as follows: 1. Data and metadata can not be synchronised across different formats, as they were not conceived for this purpose. For instance, if TIFF is selected to encode the pages of a score and AIFF is used for the corresponding performances, there is no way to keep information synchronised, unless a third format is used as a wrapper. This problem is even more evident when the user also wants to synchronise structural and musicological analyses, computer-driven performances and textual information such as lyrics. 2. Often music contents are encoded in binary form, unreadable for humans, proprietary and strongly dependent on the current technology. In our opinion, the new IEEE 1599 standard can solve the above mentioned problems. The general approach and the characteristics of this format are described in the next section. 2 IEEE 1599, a New Standard for Music IEEE 1599 is a new standard to describe comprehensively heterogeneous music contents. Its development follows the guidelines of IEEE P1599, Recommended Practice Dealing with Applications and Representations of Symbolic Music Information Using the XML Language [1]. IEEE 1599, formerly known as MX, is the result of research efforts at the Laboratorio di Informatica Musicale (LIM) of the Università degli Studi di Milano. The most recent version of its DTD and documentation can be downloaded from http://www.mx.dico.unimi.it. In a single file, music symbols, printed scores, audio tracks, computer-driven performances, catalogue metadata, text and graphic contents related to a single music piece are linked and mutually synchronised within the same framework. Heterogeneous contents are organised in a multilayered structure that supports different encoding formats and a number of digital objects for each layer [2]. Tools for music visualisation [3], content-based retrieval [4], and automatic segmentation [5] are currently available. IEEE 1599 is an XML-based format: its name is an acronym which stands for Musical application using XML. There are many advantages in choosing XML to describe information in general, and music information in particular. For instance, an XML-based language allows inherent readability, extensibility and durability. It is 274 A. Baratè, L.A. Ludovico, and A. Pinto open, free, easy to read by humans and computers, and can be edited by common software applications. Moreover, it is strongly structured, it can be extended to support new notations and new music symbols, and it can thus become a means of interchange for music with software applications and over the Net. Most of these topics have been treated in [6] and [7]. A comprehensive description of music must support heterogeneous materials. IEEE 1599 employs six different layers to represent information, as explained in [8]: • General – music-related metadata, i.e. catalogue information about the piece; • Logic – the logical description of score symbols; • Structural – identification of music objects and their mutual relationships; • Notational – graphical representations of the score; • Performance – computer-based descriptions and executions of music according to performance languages; • Audio – digital or digitised recordings of the piece. Not all layers must, or can, be present for a given music piece. For instance, the encoding of a piece does not require to describe all its structural, notational, performance or audio aspects. In some cases, even the symbolic score could not be defined, as it does not exist (such as in jazz improvisation on a harmonic grid) or it is not relevant for the user purposes (as in score following, where only the relationships between the Notational and the Audio layers must be encoded). Not only a number of heterogeneous media descriptions is supported, but each layer can also contain many digital instances. For example, the Notational layer could link to several notated versions of the score, and the Structural layer could provide many different analyses for the same piece. The concept of multi-layered description – as many different types of descriptions as possible, all correlated and synchronised – together with the concept of multi-instance support – as many different media objects as possible for each layer – provide rich and flexible means for encoding music in all its aspects. It should be clear that the description provided by such a file is flexible and rich, both in regard to the number and to the type of media involved. In fact, thanks to this approach, a single file can contain one or more descriptions of the same music piece in each layer. For example, in the case of an operatic aria, the IEEE 1599 file could house: the catalogue metadata about the piece, its author(s) and genre; the corresponding portion of the libretto; scans of the original manuscript and of a number of printed scores; several audio files containing different performances; related iconographic contents, such as sketches, stage photographs, and playbills. Thanks to the comprehensive information provided by IEEE 1599, software applications based on such a format allow an integrated enjoyment of music in all its aspects. The spine, the second key concept of the IEEE 1599 format, consists of a sorted list of events. This structure provides both an abstraction level and the glue among layers, thus representing an abstraction level, as the events identified in it do not have to correspond to score symbols, or audio samples, or anything else. It is the author who can decide, from time to time, what goes under the definition of music event, according to the needs: the definition and granularity of events can be chosen by the author of the encoding. An IEEE 1599-Based Interface for Score Analysis 275 Fig. 1. The characteristic multi-layered structure of IEEE 1599. In the right part of the figure, intuitive graphical examples are provided to illustrate the purpose of the layers. Since the spine simply lists events to provide a unique label for them, the mere presence of an event in the spine has no semantic meaning. As a consequence, what is listed in the spine structure must have a counterpart in some layer, otherwise the event would not be defined and its presence in the list (and in the file) would be absolutely useless. For example, in a piece made of n music events, the spine would list n entries without defining them from any point of view. If each event has a symbolic meaning (e.g. it is a note or a rest), is graphically rendered in many scores and is relevant to a number of analyses, these aspects are treated in the Logic, Notational, and Structural layers respectively. Music events are not only listed in the spine, but also marked by unique identifiers. These identifiers are referred to by all instances of the corresponding event representations in other layers. Thus, each spine event can be described: • in 1 to n layers; e.g., in the Logic, Notational, and Audio layers; • in 1 to n instances within the same layer; e.g., in three different score versions mapped in the Notational layer; • in 1 to n occurrences within the same instance; e.g., the notes in a song refrain that is performed 4 times (thus the same spine events are mapped 4 times in the Audio layer, at different timings). Thanks to the spine, IEEE 1599 is not a simple container for heterogeneous media descriptions related to a unique music piece. It shows instead that those descriptions can also present a number of references to a common structure. This aspect creates synchronisation among instances within a layer (intra-layer synchronisation), and – when applied to a complex file – also synchronisation among contents disposed in many layers (inter-layer synchronisation). Recondita armonia, from Tosca by G. Puccini C#5, quarter note D5, eight note E5, eight note